Optimal Randomized Approximations for Matrix-Based Rényi’s Entropy

The Matrix-based Rényi’s entropy enables us to directly measure information quantities from given data without the costly probability density estimation of underlying distributions, thus has been widely adopted in numerous statistical learning and inference tasks. However, exactly calculating this new quantity requires access eigenspectrum a semi-positive definite (SPD) matrix $A$ which grows linearly with number samples notation="LaTeX">$n$ , resulting notation="LaTeX">$O(n^{3})$ time complexity that is prohibitive for large-scale applications. To address issue, paper takes advantage stochastic trace approximations matrix-based arbitrary notation="LaTeX">$\alpha \in \mathbb {R}^{+}$ orders, lowering by converting approximation matrix-vector multiplication problem. Specifically, we develop random integer-order $ cases polynomial series (Taylor Chebyshev) fractional cases, leading notation="LaTeX">$O(n^{2}sm)$ overall complexity, where notation="LaTeX">$s, m \ll n$ denote vector queries order respectively. We theoretically establish guarantees all algorithms give explicit notation="LaTeX">$s$ notation="LaTeX">$m$ respect error notation="LaTeX">$\epsilon showing optimal convergence rate both parameters up logarithmic factor. Large-scale simulations real-world applications validate effectiveness developed approximations, demonstrating remarkable speedup negligible loss accuracy.